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CLAIMS 



[Claim(s)] 

[Claim l] An image synthesizer unit characterized by providing the following A display 
means to display an animation under photography A means to capture a frame image one 
by one from said animation, to compare two frame images with which incorporation time of 
day differs, and to calculate movement magnitude A means to memorize a subtraction 
image and said movement magnitude of said two frame images A means to create indicator 
information which symbolizes a synthetic image compounded from memorized movement 
magnitude and a captured frame image, and a means to compound a panorama image 
using said subtraction image and said movement magnitude 

[Claim 2] It is the image synthesizer unit which has a means compound a contraction 
image from a contraction frame image which said indicator information reduced said 
synthetic image, and reduced said captured frame image for a predetermined scale factor 
in an image synthesizer unit according to claim 1, and movement magnitude which 
changed said movement magnitude according to this scale factor, and is characterized by 
for said display means to display said contraction image and said panorama image. 
[Claim 3] It is the image synthesizer unit which has a means to create a symbol image in 
which said indicator information is a symbol image which symbolizes said synthetic whole 
image configuration in an image synthesizer unit according to claim 1, and a configuration 
of said synthetic image is shown, and is characterized by said display means displaying 
said symbol image and said panorama image. 

[Claim 4] An image synthesizer unit characterized by setting to an image synthesizer unit 
according to claim 2, and carrying out highlighting of the field corresponding to a frame 
incorporated by the newest in said contraction image. 

[Claim 5] An image synthesizer unit characterized by setting to an image synthesizer unit 
according to claim 3, and carrying out highlighting of the field corresponding to a frame 
incorporated by the newest in said symbol image. 

[Claim 6] Said subtraction image is an image synthesizer unit characterized by said 
panorama image composition means compounding a boundary between said two frame 
images smoothly in an image synthesizer unit according to claim 1 using this common 
image field also including a common image field of said two frame images. 
[Claim 7] It is the image synthesizer unit characterized by calculating a pixel of this 
expanded subtraction image with interpolation when expanding said subtraction image 
and creating a synthetic image in an image synthesizer unit according to claim 1. 
[Claim 8] An image synthesizer unit characterized by providing the following. A display 
means to display a photoed animation A means to capture a frame image one by one from 
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said animation A means to compare two frame images with which incorporated time of day 
differs, and to calculate movement magnitude of an image A means reduce this synthetic 
image remembered to be a means obtain a synthetic image combining a frame image 
captured based on said movement magnitude, a buffer means memorize said synthetic 
image or its contraction image, and a preview display means display one field of said 
synthetic image memorized by said buffer by said buffer when said synthetic image or its 
contraction image exceed a predetermined field, and said one field 

[Claim 9] A record medium with which an image composition method characterized by 
providing the following was recorded and in which a computer readout is possible A step 
which displays an animation under photography A step which captures a frame image one 
by one from said animation, compares two frame images with which incorporation time of 
day differs, and calculates movement magnitude A step which memorizes a subtraction 
image and said movement magnitude of said two frame images A step which creates 
indicator information which symbolizes a synthetic image compounded from memorized 
movement magnitude and a captured frame image, and a step which compounds a 
panorama image using said subtraction image and said movement magnitude 
[Claim 10] A step which compares two frame images with which incorporated time of day 
differs, and calculates movement magnitude of an image, A step which obtains a synthetic 
image combining a frame image captured based on this movement magnitude, A step 
which memorizes this synthetic image or its contraction image to a buffer, A step which 
displays one field of this synthetic image memorized by this buffer and which indicates by 
preview, A record medium with which an image composition method by which it is having 
[ when this synthetic image or its contraction image exceeds the field range of 
regular ]*step which reduced and displays this synthetic image and this one field which 
were memorized by this buffer characterized was recorded and in which a computer 
readout is possible. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[The technical field to which invention belongs] This invention relates to the image 
synthesizer unit which compounds a panorama image on real time from the dynamic 
image which is applied to the record medium with which the synthesizer unit and the 
synthetic method of compounding the panorama image of a wide-field"of-view angle were 
recorded, especially continues, and this record medium by which synthetic method drawing 
record was carried out. 
[0002] 

[Description of the Prior Art] Two or more images which photoed a different visual field are 
compounded, and the technology which makes the panorama image of the wide field of 
view of one sheet attracts attention. Especially recently, interests have gathered for the 
technology which creates a panorama image similarly from the continuation dynamic 
image photoed while carrying out the pan of the video camera and changing the visual field 
and which is called Video Mosaic. For example, Teodosio of U.S. Massachusetts Institute of 
Technology, L. Bender, W. It depends. Salient Video Stills: Content and Context Preserved, 
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Proc. ACM Multimedia 93, pp. 39-46 (1993) etc. - it is known. In addition, although 
various examples of research exist, the panorama image has been obtained by sticking an 
image so that the common area may be found and a common area may lap exactly about 
the image of two or more sheets with which any technique was acquired continuously. 
Since, as for close, the image of a new visual field comes to a photography image by 
following on moving a camera, they are connected and, finally the panorama image of the 
wide field of view is obtained. 

[0003] However, in many examples of research, things for which the precision of a 
synthetic image is raised, such as how such a common area is found with a sufficient 
precision or how to amend a gap of the image periphery by distortion of a lens, are thought 
as important, and it is hard to say that the examination from a viewpoint of the usabiliity 
of the operator using such panorama composition technology is actually enough. In order to 
perform image recognition generally about the natural image which cannot specify the 
class of target image, human being needs to intervene and it is necessary to give suitable 
decision on various aspects of affairs. In this case, the high-speed responsibility of 
processing influences an operator's user-friendliness greatly, depending on a use, it 
connects that there is no breakdown as an image — even having — if it is, it is not few also 
when the technique excellent in usabiliity which can process an input image without delay 
to the input of real time, i.e., an image, rather, and the result can check immediately is 
more desirable. 
[0004] 

[Problem(s) to be Solved by the Invention] Two kinds of processings need to be performed 
in creation of a panorama image. One is presumed processing of the amount of camera 
motions in which recognize a continuous input image and the coordinate of a common area 
is determined. Another is obtained synthetic processing which moves and actually connects 
an image from an amount. It is most desirable to perform these to coincidence on real time 
moreover from a viewpoint of usabiliity. That is, when the shaken amount is calculated, an 
image ties, and is set automatically and it finishes photographing, it is an ideal that the 
synthetic image is already made and it can check, at the same time it shakes a camera. 
Redo can be done freely any number of times, even if it adopts how with missing 
information to essentially photograph an uncompoundable image, on that spot, a mistake 
is known and the fundamental correction from how to photograph can be performed, until 
the image included in mind is obtained by this. 

[0005] When the shaken amount is calculated, an image ties, and is set automatically, and 
it finishes photographing, and the synthetic image is already made and it enables it to 
check at the same time it shakes a camera, both these two kinds of presumed processings 
and synthetic processing need to process on real time. About presumed processing of the 
amount of camera motions, there is high-speed mode of processing shown by 
JP,11-004398,A "a digital wide camera" by artificers, for example. However, about 
synthetic processing, it cannot be said that even the level which can be performed on real 
time at coincidence fully inquires. After photography, a synthetic result cannot be checked 
but it is inadequate that the amount of motions is just calculable on real time to 
photography and coincidence in the field of usabiliity, if it does not wait for the time 
amount of lamination processing of images, although a calculated part of the amount of 
motions becomes short. 

[0006] The purpose of this invention is to realize high-speed responsibility of the synthetic 
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processing which can check a synthetic result on real time in order to raise an operator's 
usabiliity in the method of compounding a panorama image from a continuous dynamic 
image. 
[0007] 

[Means for Solving the Problem] A display means to display an animation under 
photography in order to attain the above-mentioned purpose, and a means to capture a 
frame image one by one from said animation, to compare two frame images with which 
incorporation time of day differs, and to calculate movement magnitude, A means to record 
a subtraction image and said movement magnitude of said two frame images, A means to 
create indicator information which symbolizes a synthetic image compounded from 
memorized movement magnitude and a captured frame image, and a means to compound a 
panorama image using said subtraction image and said movement magnitude are 
established. 
[0008] 

[Embodiment of the Invention] Hereafter, one example of this invention is explained to 
details. Drawing 1 is an example of the outline block diagram of the system configuration 
for realizing this invention. 100 is the general video camera image pick-up section, and 
transmits the photoed image to a processing unit 116 through the video signal line 106. 
The camera section is connected with a processing unit 116 by the control signal line 108, 
and 116 has come to be able to perform a setup of the camera parameter of being as 
adjusting a white balance **** [, and ], if needed. [ changing the zoom scale factor of a 
video camera ] It can omit, when such a setup is unnecessary. The processing unit 116 is 
the same as the system configuration of the digital computer used for the current 
general-purpose target, especially a personal computer, the display output is drawn on the 
screen of a display 112 through a video signal line, and carrier beam directions are notified 
to a processing unit 116 by the input device 120 from a user. Serially, the video signal 
outputted from the camera section 100 is changed into digital image data by the A/D 
converter of 102, goes into memory 114 through an interface 104, and is processed by 
CPU110 according to the program stored in another address of memory 114. 
[0009] Here, although the signal from a video camera explains in the example in the case of 
analog video format, such as NTSC, when using the video camera of a digital output, the 
A/D converter of 102 is unnecessary. In this case, the converter of image format may be 
instead needed. Moreover, according to the necessity for processing, various information 
can be accumulated in the external information storage device of 118. 

[0010] In case various kinds of data created by the processing explained below is stored in 
memory 114 and CPU 110 processes, it is referred to if needed. Although the following 
explanation explains as software which CPU performs altogether, the part or all may be 
replaced and performed by the hardware logic circuit. Moreover, you may be the intelligent 
video camera gestalt which unified the camera section 100, the processing unit 116, and 
the display 112. 

[00 11] Below, the software flow which is performed on the above-mentioned hardware and 
which solves the technical problem of point ** is explained to details. First, the creation 
method of the panorama image using the above-mentioned software is explained. A user 
tells shifting to the condition (it being called panoramic exposure mode below) that 
panorama composition is first performed on real time to a processing unit 116 with the 
input device 120. When an input device uses a computer system, in taking a camera one 
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apparatus gestalt in a keyboard, a mouse, etc., switches, such as a "panoramic exposure 
initiation carbon button", correspond. If a processing unit 116 goes into panoramic 
exposure mode, it will carry out the ** frame input of the video signal outputted from a 
video camera, will calculate the amount of camera motions of the inputted frame image, 
and will collect parameters and data required for panorama composition. At this time, that 
whole or some fields are set up by the display 112 as a preview screen for checking a 
synthetic condition, and the appearance of the synthetic image connected according to the 
motion of a camera is indicated by updating on real time at it. 

[0012] The image is shown in the display 112 of drawing 1 . In this invention, the synthetic 
image 122 (it is hereafter called a preview image) for a preview display is characterized by 
always carrying out a scaling and displaying that it becomes the greatest magnitude in 
which the whole fits in a preview screen. By this, the grasp at an instant of the whole 
synthetic condition is enabled. This is fill uped with the suitable background color, 
although the null portion 124 in which an image does not exist exists with the 
configuration of a synthetic image when a preview screen is a rectangle at this time. The 
location of the newest input frame in a synthetic image is surrounding, or changing a color 
and carrying out highlighting by the frame, and the role of the compass which does not 
puzzle the direction from which a camera should be moved is also given (126). 
[0013] Again, with the input device 120, if a user directs termination in panoramic 
exposure mode, he stops a new frame image input and will be in a standby condition. The 
preview image updated by the last frame input by the display 112 is held, and a user looks 
at the image, judges whether the result of a request is obtained, if good, he creates the 
synthetic image (it is hereafter called a final output composition image) as a final output of 
the real size which is not an object for a preview, and saves at the external information 
recording device 118 etc. 

[0014] An example of a flow chart which realizes the above-mentioned function to drawing 
2 is shown. This algorithm makes the following three points the basic concept in order to 
enable high-speed processing indispensable to the above-mentioned function. 
[0015] (l) Create the synthetic image of contraction size only to previews apart from a final 
output composition image. 

[0016] (2) Let the image memorized in the last composition images be only an 
indispensable field. 

[0017] (3) Reconcile creation of the preview image of an arbitration configuration and 
arbitration size, and memory effectiveness. 

[0018] (l) is a device for reducing the time amount concerning synthetic processing. It is 
not necessary to be the image of size with the created image not necessarily big generally, 
in order to judge whether it is a desired thing. So, at the time of photography, a final 
output composition image is not created but creates the synthetic image of contraction size 
separately as an object for a preview. It is made for a play view image to be mostly 
completed to coincidence with photography by this. Since composition is performed based 
on the camera motion estimate from which this preview image was obtained, near, 
distortion, a mistake, etc. are extremely reflected in an actual synthetic result as it is. If a 
user judges whether the desired image was obtained and is checked by this preview with 
right **, he will create the final output composition image of actual magnitude anew. Since 
it can redo immediately and the time amount of composition of real size can be saved when 
it is judged as failure, an efficient activity can be performed. 
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[0019] (2) is a device for reducing the chart lasting time of the former data for a final 
output composition image. When (l) is adopted and a final output composition image is 
made to be created later, it is necessary to record temporarily the REMU image which is 
the former data. It becomes the inhibition factor of the high-speed processing by it being as 
that required memory space ****** s **** [ 9 and ] to record the new frame image inputted 
one after another one after another. [ that it takes a data transfer time for a long time ] 
[0020] Here, as shown in drawing 3 , a panorama composition image piles up every sheet of 
a continuous frame image, and is obtained. Drawing 3 is an example which creates a 
panorama image from a series of panimages, expresses typically the image which consists 
of a frame image with which the bottom continues, and shows the panorama image with 
which the bottom is compounded from it. Two or more lines which tie up the upper and 
lower sides express the correspondence relation of frame images. By the lower panorama 
image, the continuous field (it is hereafter called a common area) which is in agreement 
always in image by inter-frame exists, and redundancy is very high as data. Then, a 
panorama image is fully reproducible by combining only the field of the portion which 
excluded the this continuation inter-frame common area. Therefore, if only the field which 
excluded this common area per frame is recorded, the amount of data which should record 
per frame is reduced sharply, a data transfer time will be shortened and its required 
memory space will also decrease. Although the whole image is needed only for 
superposition of the last frame, since it is only the 1 time uniquely, and it is the last and 
the following frame does not exist even if time amount cuts some in many to data transfer, 
the problem of not being of use for the processing for degree frame is not generated, either. 
[0021] Although it is about (3) of the last, a preview image conforms to a motion of a 
camera out of photography further, it is not only obtained by photography termination and 
coincidence, but is updated synchronizing with a motion of real time, i.e., a camera, and a 
check is [ way ] easy for a user and it is user-friendly. However, the panorama image 
compounded becomes large in monotone as it moves a camera, and the configuration and 
magnitude become various depending on how to move a camera. In order to display a 
preview image, the panorama image compounded for a preview must always be created on 
the memory area, but in order to create the possible preview image of all configurations 
and magnitude on a memory area, a beforehand very large memory area must be taken. 
[0022] Though a contraction frame image is used with the indicator of (l), the capacity of a 
required memory area is huge. And there will also be many memory portions which are not 
used in fact, and, as for an emergency, the utilization ratio of memory will become low. Of 
course, the direction from which a camera is moved first can be determined and only a part 
required for it can secure a memory area. However, by this method, even if the 
complicatedness which must show the direction from which a user moves a camera 
beforehand arises, furthermore mind changes a user on the way, it cannot respond at all. 
Moreover, finally it becomes the image of the magnitude of which, or a user cannot predict 
in many cases. To correspond to the synthetic image of unrestricted size and an 
unrestricted configuration is desired. 

[0023] Here, the magnitude of the preview screen for displaying a preview image on a 
monitor is immobilization, and when the magnitude of a preview image becomes larger 
than the preview screen of the magnitude of immobilization, a contraction scaling is 
carried out and it is displayed to suit the screen size. Conversely, if it says, the number of 
pixels exceeding the resolution of a preview screen will become useless in case a preview 
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image is displayed. 

[0024] Then, the magnitude of a preview image is n times (length or ratio as horizontal 
length.) of the magnitude of a preview screen. When nxn times are exceeded in area, the 
magnitude of a preview image is compulsorily reduced by 1/m time (similarly in area 
l/(mxm) twice). And when the magnitude of the reduced preview image exceeds n times of 
a preview screen again, it repeats as 1/m time contraction of a preview image is performed. 
The effect that the preview image which can restrict the memory area size for preview 
images to n or less times of the size of a preview screen, and is displayed on a preview 
screen by this does not have deterioration of resolution compared with the case where it 
has a memory area beyond it is acquired. 

[0025] Based on the above thing, it returns to explanation of the flow chart of drawing 2 . 
Various kinds of initialization is first performed by processing 200. For example, the 
memory area (it is hereafter called a preview buffer) of the magnitude which carried out 
magnitude of a preview screen in all directions [ n times as many as this / each ] is secured 
as a working buffer for preview image creation. A preview image is compounded in this 
preview buffer, the field called a preview viewing area is set to a preview buffer, and this 
field fills the magnitude of a preview screen exactly - as always - a scaling - that is, 
zooming is carried out and it becomes the display of a preview screen. 

[0026] A sequential preview viewing area is expanded so that the whole preview image 
may always be contained, as the field of the same magnitude as the preview screen of a 
buffer center section is set up as a preview viewing area at first and a preview image 
becomes large on a preview buffer. Subsequently, the initial value of the preview reduction 
percentage r which is a variable for preview image creation is calculated (202). r is the 
ratio of the magnitude of the frame image inputted from a camera, and the magnitude of a 
preview screen, and is a parameter adjusted so that an input frame image may be 
displayed to the limit of a preview screen in an initial state without a motion of a camera. 
For example, it is r= 1 when magnitude of an input frame image and magnitude of a 
preview screen are made the same. It is set to 0. 

[0027] In the continuing processing 204, the frame image of the one newest coma of the 
image photoed with the video camera is inputted. And movement magnitude detection 
processing of an image is performed, comparing between the frame images inputted one 
after another. Although this movement magnitude detection considers various methods, 
such as an optical flow, the method indicated by the "digital wide camera" of Japanese 
Patent Application No. 9*153303 is used for it as a method computable at a high speed here. 
This method enables high-speed processing very much, in order to calculate movement 
magnitude from projection distribution of an image. 

[0028] In processing 206, the level projection distribution Px and the perpendicular 
projection distribution Py are created from this frame image. Level projection distribution 
is what took the average of the color and brightness about each line of the pixel which 
constitutes an image, and is the single dimension sequence of the average only for a line 
count. Perpendicular projection distribution is what took the average of a color or 
brightness similarly about each train of an image, and is the single dimension sequence of 
the average only for the number of trains. For example, the list of the value of 
perpendicular projection distribution has the property shifted to the forward direction or 
hard flow corresponding to it, when an image moves to right and left. Perpendicular 
projection distribution is similarly shifted to order hard flow corresponding to vertical 
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movement of an image. Therefore, the horizontal / perpendicular projection distribution 
created from a certain frame image are compared with the horizontal / perpendicular 
projection distribution created from the frame image inputted into the degree, and if it 
asks for which the sequence of the average shifted, respectively, the movement magnitude 
of level/perpendicular direction between the images of two sheets can be found. Although 
the list of the value of projection distribution is completely in agreement with neither a 
noise nor some deformation since it is image change, one of the two of two projection 
distribution to compare is shifted by delta pixel one by one, and is compared (all integers 
for which delta takes the range of -R to +R.). R obtains the value which shows the retrieval 
range set up beforehand, and delta when dissimilarity becomes low most as movement 
magnitude. Dissimilarity uses absolute total of the difference of the color of the pixel of a 
corresponding line or a train, or each average of brightness etc. (210*212). However, since 
the frame image in comparison with the past does not exist in the first image input, it 
returns to processing 204 (208). 

[0029] In processing 214, from obtained level / the amount delta of vertical migration, it 
asks for the common area of the last frame image and the present frame image, and 
records only about the field of portions other than the common area in the last frame. By 
this, there is little storage capacity compared with the case where it records about all fields, 
and it ends. 

[0030] Moreover, in processing 216, the movement magnitude for preview composition 
which multiplied the value of the obtained horizontal / the amount delta of vertical 
migration by r is calculated, and the image reduced by r times is drawn in piles in the 
location which shifted only deltaxr obtained above to the contraction frame image which 
drew on the preview buffer immediately before. Thus, the preview viewing area updated so 
that the expanded whole preview image might be included is displayed on a preview screen 
(218). 

[0031] At this time, when the whole preview image is as Hasama to the limit of the edge of 
a preview buffer, (220) and the whole preview buffer are reduced by 1/m time (222). At this 
time, a preview viewing area is similarly reduced by 1/m time. And r multiplies by 1/m 
similarly. 

[0032] The typical flow of contraction processing of this preview buffer is shown in drawing 
4 . 400 shows the whole preview buffer and 402 is a preview viewing area. 404 shows a 
preview image. A location and magnitude change so that a preview viewing area may 
always be circumscribed to the whole preview image. A expresses the situation 
immediately after panoramic exposure initiation, and the inputted frame image is drawn 
by the center section. The preview image is extended, also follows a preview viewing area, 
and becomes large as a camera is moved (B). And since a buffer will be overflowed if a 
preview image reaches to the edge of a preview buffer as shown in C, the whole preview 
buffer is reduced by 1/m time here (D). At this time, a preview viewing area is also reduced 
for the same scale factor. And if a preview image is extended further, drawing of an image 
will be continued according to it. 

[0033] Here, if n= 3 and m= 2, first, the center of a preview buffer and the center of a 
viewing area will be doubled, and the size of the preview viewing area when arriving at 
elongation and an edge after that according to a photography image will become always 
larger than the twice (a length ratio in every direction is twice) of the original size of a 
preview screen, therefore, this - every the whole preview buffer 1/2 twice (it is a length 
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ratio in every direction 1/2) - even if the size of a preview viewing area is not less than 
the original size of a preview screen In addition, when a preview viewing area reaches that 
of a buffer exactly at the time of n= 3 and m= 2, a preview viewing area becomes twice the 
original size of a preview screen. Then, if a preview buffer is doubled 1/2, the size of a 
preview viewing area will become the same as the original size of a preview screen. 
[0034] The result which hardly changes to the case where secure a huge memory area, 
draw a preview image freely, and it thins out and displays on the size of a preview screen 
on it by this will be obtained only by preparing n times as much memory space as a preview 
screen. Moreover, since what is necessary is just to thin out at intervals of a pixel simply in 
the case of m= 2, contraction can also be extremely performed at a high speed. 
[0035] Here, it considers creating a panorama image 10 times the magnitude for example, 
of an input frame image. By the method of securing a memory area beforehand, without 
being based on the above-mentioned method, in order to preview the panorama image of 
this area, an about 400 times as many memory area as an input frame image is needed 
beforehand. Since this cannot predict in which direction a camera moves from the 
beginning, it is because it is necessary to secure the rectangle field which has one 10 times 
the length of this in each vertical and horizontal direction centering on the first input 
frame image, respectively. However, according to the method of this invention, it is [ at one 
at most 9 times the magnitude of this ] good in the case of n= 3. Since this invention 
guarantees unrestricted size by one 9 times this magnitude of this, the difference of a 
memory area with the conventional method is followed on the magnitude of the panorama 
image to compound increasing 20 times and 30 times, and becomes large exponentially. 
[0036] Furthermore, in high resolving still picture composition processing in which the 
zoom image was used, the above-mentioned technique has an effect further. The principle 
of the high resolving still picture composition by the zoom is shown in drawing 5 . This 
fundamental view is already mentioned reference Salient Video Stills. • It is shown in 
Content and Context Preserved. A big high resolution image can be obtained by making 
the frame image when zooming in most into criteria size by piling up other frame images 
according to the rate of a zoom. This high resolution image becomes highly minute little by 
little gradually as a periphery approaches a core, although a pixel only becomes large 
simply. The periphery is fading and its image photoed with the usual still camera is also 
close to the property of the high resolution image by which superposition composition was 
carried out, when a focus is doubled with a center. That is, an image equivalent to having 
copied using the image sensor of expensive high resolution will be obtained using the 
image sensor of a cheap low resolution. 

[0037] However, by such conventional technique, the expansion processing time also 
becomes long and a real-time operation becomes difficult further rather than the time of 
panorama composition as superposition processing must be performed and a dilation ratio 
becomes large, after expanding the image at the time of zoom out. What is necessary is just 
to draw the first frame by actual size, and to pile up the rest, expanding a frame image, if a 
huge memory area can be prepared first in the case of an image which carries out zoom out 
in monotone still gradually. However, in the case of an image which zooms in reverse, it is 
necessary to expand the synthetic image, and time amount requires expanding a huge 
memory area the whole ** frame very much so that the frame image which is the purpose 
of high resolving image composition and into which resolution was inputted at the end by 
composition with slight height may always become actual size. This is the technical 
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problem of the real-time operation no one but which cannot predict any motion [ what kind 
of camera ] there is beforehand. 

[0038] On the other hand, if the view of this invention is applied, a preview can be checked 
on real time in few memory areas. About the technique of asking a high speed for the rate 
of a zoom, the technique already shown in Japanese Patent Application No. 10-53100 by 
artificers "a super resolution camera" can be used, for example. When it checks here, it is 
piling up the frame image inputted, adjusting size so that it may become the contraction 
scale as composition of a high resolving image with the same common area between 
continuous frame images. Therefore, like the time of panorama composition, in order to 
obtain a final synthetic image, a common area requires only the character type outer frame 
portion [ in / in the case of the image of zoom-in / the last frame ] of RO except a common 
area that what is necessary is to record only once. Moreover, what is necessary is just to 
merely draw the frame image reduced according to the rate of a zoom one after another to 
the preview buffer in creation of a preview image. 

[0039] The rate of a zoom is defined here as a ratio when carrying out size doubling so that 
a common area may be in agreement on the basis of the frame image inputted first. It is 1 
when there is no zoom. In 0 and zoom-in, it is smaller than 1, and, in the case of zoom out, 
it becomes large from 1. On the other hand, in the case of the image of zoom out, since the 
range where the newest frame is larger is copied, the character type outer frame portion of 
RO except the common area in the newest frame is recorded shortly. If creation of a 
preview image hits, this outer frame portion is reduced according to the rate of a zoom, and 
it draws. If the rate of a zoom becomes [ the size of a preview image ] large, since the size of 
a preview buffer will be exceeded in the case of zoom out, the whole preview buffer is 
reduced to 1/m like the time of panorama composition in that case. Thus, an expansion 
required for preview image creation also in the case of a zoom is only processing to which 
only the character type outer frame portion of RO is expanded to n or less times, and can be 
clearly processed at a high speed. By the above, the result of high resolving image 
composition can also be checked on real time to photography and coincidence. 
[0040] An example of the former data recorded in this example for a final synthetic image 
is shown in drawing 6 and drawing 7 . Data is divided roughly into the panorama file 
structure object 600 and the panorama clip structure 700. The former panorama file 
structure object is a header which gathers the whole former data, and the latter panorama 
clip structure is image data as which record is determined for every frame. The real size of 
a file ID 602 and a synthetic image is stored in a panorama file structure object in order of 
width of face (604) and height (606) as an identifier which shows that consecutiveness data 
is former data for image composition. Since the final size of a synthetic image can be 
expressed as the minimum rectangle circumscribed to it even if it was determined from the 
amount of camera motions and has taken the special configuration, only width of face and 
height are stored. Moreover, the color number 608 (it is one full color 24 bits etc. about 
whether they are for example, 256 colors) and the graphics format 610 (for example, it is 
the YUV format about whether it is RGB format) of former data are also stored. 
[0041] Moreover, the starting point coordinate of the rectangle field circumscribed to a 
synthetic image is stored as the panorama image x and y offset (612 614), and the 
coordinate of the panorama clip later mentioned on the basis of this coordinate is 
calculated. The panorama clip structure 700 is located in a line as data following this 
header information. The panorama clip structure serves as data for every one frame, and 
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stores the location and size of the frame image in a synthetic image. First, 702 is the total 
data size and stores the amount of image data. The image starting point x-coordinate 702 
expresses the x-coordinate value of the starting point of the frame image matched with this 
panorama clip structure when making into the starting point coordinate of a synthetic 
image the coordinate shown by the panorama image x above-mentioned offset 612. 
Similarly, the image starting point y-coordinate 704 expresses the y-coordinate value of 
the starting point of a frame image. The image width of face 706 and the image high 708 
express real [ that a frame image is inputted ] size. The size of the frame image on a 
synthetic image is calculated by multiplying this by the rate 710 of a zoom. 714-720 specify 
the range of the above-mentioned common area in this frame image. The common area 
starting point x-coordinate 714 and the common area starting point y-coordinate 716 
express the starting point coordinate of the common area in a frame, are the common area 
width of face 718 and common area high [ 720 ], and express the magnitude. All pixel data 
is located in a line with the pixel data 722 in order, scanning in the direction of the lower 
right from the upper left. When the suitable value for 714-720 is in close, the pixel data 
about a common area is disregarded simply. 

[0042] Next, how to create a final synthetic image from the former data memorized above 
is explained. With reference to the panorama file structure object 600, the width of face 
and the height of a synthetic image are investigated first, and the rectangle image field of 
the magnitude is secured in a memory area. And with reference to the panorama clip 
structure 700 which follows, the location and magnitude which should draw are 
investigated, it asks for the relative position in the secured memory area, and image data 
722 is written in. However, when at least one clip whose rate of a zoom is less than 1 time 
exists, proportionality conversion of the rate of a zoom of all the panorama clip structures 
is carried out so that the smallest rate of a zoom may be first set to 1, namely, so that the 
frame image which zoomed in most may be drawn by actual size. In this case, the size of a 
synthetic image also becomes large proportionally. It is lost by this that a frame image is 
reduced and composition is performed only in the direction in which resolution is raised. 
[0043] Smoothing processing can be performed in order to show more beautifully the image 
composition described above. Generally, when sticking an image, there may be distortion 
in the time of the input image according to the aberration of a lens in that the amount of 
motions cannot be found correctly ****, or the brightness in a photography time may be 
changed, and a delicate boundary line may occur in the knot of an image. With human 
being's perception, this boundary line needs to amend in order to give an unpleasant 
impression. The easiest method of performing such amendment is sticking so that an 
edge ! s left for applying paste may be given among the images to stick, a boundary line's 
may be inserted and it may change to the color of the image of another side gradually from 
the color of one image. The image which was simplified very much and showed the concept 
is shown in drawing 8 . 800 shows signs that the image of two sheets is stuck simply. Here, 
in order to simplify explanation, the image of two sheets assumes that it is uniform by 
specific brightness, respectively. On the boundary line of an image, if change of the 
horizontal brightness at this time is looked at, as shown in a graph 802, it will change in an 
instant and this will form a boundary line 808. On the other hand, as shown in 804 and 806, 
if it sticks so that it may become a gradual change from the brightness of one image to the 
brightness of the image of another side, a boundary line will not be conspicuous [ an edge 
left for applying paste is given to the boundary of an image, and ]. 
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[0044] In order to realize the above-mentioned smoothing processing, when recording 
former data by the synthetic method of this invention, only the part of an edge left for 
applying paste should record a larger field. By the increment in the necessary minimum 
amount of data, smoothing becomes possible by this. And the synthetic image connected 
smoothly is obtained by performing smoothing processing about the part of an edge left for 
applying paste in the case of next synthetic processing. The field made into an edge left for 
applying paste is chosen in the form which narrows the common area between the 
continuous images of two sheets fundamentally. The example in the case of having 
arranged so that a common area may lap with drawing 9 about the two frame images 900 
and 902 is shown. Supposing 902 is a new frame image in time, the L character mold 
hatching field 904 at the upper left of drawing will be recorded as former image data. Since 
an edge left for applying paste should just bury the boundary line of two images gradually, 
it serves as a field which needs the field which swelled width of face for the direction which 
narrows a common area in accordance with a boundary line. In the example of drawing 9 , 
the field used as an edge left for applying paste turns into the hatching field 906 of the L 
character mold inside 904. It is shown in a list at drawing 10 by carrying out combination 
of the typical amount of camera motions, and the edge-left-for-applying-paste field at that 
time, deltax and delta y show the movement magnitude of level/perpendicular direction 
among drawing, respectively. The pattern of the same edge-left-for-applying-paste field in 
case a zoom is further shown in drawing 11 is shown. By a diagram, the case of zoom-in is 
shown. Thus, even if there is a zoom, from the boundary line between the images when 
doubling magnitude so that the magnitude of a common area may become the same, it 
extends in the direction of a common area, and an edge-left-for-applying-paste field is 
decided. Also in zoom out, it is the same only by the relation to reverse of the frame images 
900 and 902 becoming. 

[0045] In addition, when creating a highly minute image, in case image data is stuck 
according to the rate of a zoom, it cannot be overemphasized that a still better-looking 
image is obtained by expanding using pixel interpolation technology, such as bilinear 
interpolation instead of expansion of a simple pixel. And a still more beautiful image is 
obtained by performing the above-mentioned smoothing processing for the image data 
expanded with pixel interpolation. 

[0046] Moreover, although the technique of creating and displaying a preview image on 
real time was described above, as it indicates by simple instead of a preview image as the 
symbol which shows the situation under composition, coexistence with usabiliity and much 
more improvement in the speed can also be aimed at. For example, since it is only a line 
drawing, drawing becomes possible at a high speed, at the same time necessity is also lost 
by reducing and sticking a frame image by showing with a line drawing whether not the 
preview image itself but the silhouette, i.e., a synthetic image, is made of what kind of 
configuration. The example of a preview screen based on this is shown in drawing 12. On a 
preview screen, an input image is displayed as it is, and an indicator indication also of the 
line drawing 1200 is given together with it on it. Moreover, in order to show where [ in a 
synthetic image ] the frame under current input hits, as shown in 1202, highlighting of the 
field which a frame image current in the inside of a line drawing occupies may be carried 
out. The purpose of checking a synthetic condition on real time is realized more by the 
operation engine performance of low cost by this. 
[0047] 
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[Effect of the Invention] According to this invention, in creation of a panorama image or a 
high resolving image, the outstanding usabiliity which can be retaken can be freely offered 
any number of times until the image which could check the advance condition of processing 
and the result to photography and coincidence, and went into them at mind is obtained. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing l] It is an example of system configuration drawing for realizing this invention. 
[Drawing 2] It is an example of the flow chart of the processing for realizing this invention. 
[Drawing 31 It is image drawing which compounds a panorama image from the animation 
which carries out a pan. 

[Drawing 4] It is an example of the contents of the preview buffer at the time of performing 
this invention. 

[Drawing 5] It is image drawing which compounds a high resolving image from the 
animation which carries out a zoom. 

[Drawing 6] It is an example of the data structure used by processing of this invention. 

[Drawing 7] It is an example of the data structure used by processing of this invention. 

[Drawing 8] It is image drawing showing the image lamination method. 

[Drawing 9] It is drawing explaining each image field which processing of this invention 

defined. 

[Drawing 101 It is an example of the pattern of an edge*left-for*applying-paste field used by 
processing of this invention. 

[Drawing 111 It is an example of the pattern of an edge-left-for-applying-paste field used by 
processing of this invention. 

[Drawing 12] It is an example of an indicator display used by processing of this invention. 
[Description of Notations] 

100 [ - A video signal line, 108 / - A control signal line, 110 / - CPU, 112 / - A display, 114 
/ - Memory, 116 / - A processing unit, 118 / - An external information storage device, 120 / 
- Input device. ] - The camera section, 102 - An A/D converter, 104 - An interface, 106 
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